{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a6199b61-8084-4a2f-8488-2ddc4c260776",
   "metadata": {},
   "source": [
    "# Vertex AI\n",
    "\n",
    "Vertex AI offers everything you need to build and use generative AI—from AI solutions, to Search and Conversation, to 100+ foundation models, to a unified AI platform. You get access to models like PaLM 2 which can be used to score your RAG responses and pipelines with Ragas instead of the default OpenAI.\n",
    "\n",
    "This tutorial will show you can you can use PaLM 2 with Ragas for evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79986e4d-17e4-4e88-ad2e-ca2b34a63de9",
   "metadata": {},
   "source": [
    ":::{Note}\n",
    "this guide is for folks who are using Google VertexAI endpoints. Check the [evaluation guide](../../getstarted/evaluation.md) if your using OpenAI endpoints.\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bd757ae-51ee-4527-b2fb-0d0b88285939",
   "metadata": {},
   "source": [
    "## Load Sample Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0d3e6c99-c19c-44a1-8f05-4bde2de30866",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset amnesty_qa (/home/jjmachan/.cache/huggingface/datasets/explodinggradients___amnesty_qa/english_v2/2.0.0/d0ed9800191a31943ee52a5c22ee4305e28a33f5edcd9a323802112cff07cc24)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f19a3848cdcc4c08b8a52cde89cab643",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    eval: Dataset({\n",
       "        features: ['question', 'ground_truth', 'answer', 'contexts'],\n",
       "        num_rows: 20\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data\n",
    "from datasets import load_dataset\n",
    "\n",
    "amnesty_qa = load_dataset(\"explodinggradients/amnesty_qa\", \"english_v2\")\n",
    "amnesty_qa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e67daaa-60e3-4584-8ec6-944c3c5a1a0c",
   "metadata": {},
   "source": [
    "Now lets import the metrics we are going to use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "26732db3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.metrics import (\n",
    "    context_precision,\n",
    "    answer_relevancy,  \n",
    "    faithfulness,\n",
    "    context_recall,\n",
    "    answer_similarity, \n",
    "    answer_correctness,\n",
    ")\n",
    "from ragas.metrics.critique import harmfulness\n",
    "\n",
    "# list of metrics we're going to use\n",
    "metrics = [\n",
    "    faithfulness,\n",
    "    answer_relevancy,\n",
    "    context_recall,\n",
    "    context_precision,\n",
    "    harmfulness,\n",
    "    answer_similarity,\n",
    "    answer_correctness,\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20253a48",
   "metadata": {},
   "source": [
    "By default Ragas uses `ChatOpenAI` for evaluations, lets swap that out with `ChatVertexAI`. We'll wrap `ChatVertexAI` with Ragas' `LangchainLLMWrapper` object to work with the `langchain-google-vertexai` package. We also need to change the embeddings used for evaluations for `OpenAIEmbeddings` to `VertexAIEmbeddings` for metrices that need it, which in our case is `answer_relevancy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "09ef783b-12dd-40e8-bdbf-4744e41038dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import google.auth\n",
    "from langchain_google_vertexai import ChatVertexAI, VertexAIEmbeddings\n",
    "from ragas.llms import LangchainLLMWrapper\n",
    "\n",
    "config = {\n",
    "    \"project_id\": \"<your-project-id>\",\n",
    "    \"chat_model_id\": \"gemini-1.0-pro-002\",\n",
    "    \"embedding_model_id\": \"textembedding-gecko\",\n",
    "}\n",
    "\n",
    "# authenticate to GCP\n",
    "creds, _ = google.auth.default(quota_project_id=config[\"project_id\"])\n",
    "\n",
    "# create Langchain LLM and Embeddings\n",
    "vertextai_llm = ChatVertexAI(\n",
    "        credentials=creds,\n",
    "        model_name=config['chat_model_id'],\n",
    "    )\n",
    "vertextai_embeddings = VertexAIEmbeddings(\n",
    "    credentials=creds,\n",
    "    model_name=config[\"embedding_model_id\"]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27a34b9d-796e-4025-ba8e-a799f6ab3c6b",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "Running the evalutation is as simple as calling evaluate on the `Dataset` with the metrics of your choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5e739223-4e34-4dc0-9892-4625fdea7489",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "26118d5ea3a544e5a136e8716b12f3ab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Evaluating:   0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'faithfulness': 0.9583, 'answer_relevancy': 0.8608, 'context_recall': 1.0000, 'context_precision': 1.0000, 'harmfulness': 1.0000, 'answer_similarity': 0.9405, 'answer_correctness': 0.3757}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from ragas import evaluate\n",
    "\n",
    "result = evaluate(\n",
    "    amnesty_qa[\"eval\"].select(range(1)),  # using 1 as example due to quota constrains\n",
    "    metrics=metrics,\n",
    "    llm=vertextai_llm,\n",
    "    embeddings=vertextai_embeddings,\n",
    ")\n",
    "\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "960f88fc-c90b-4ac6-8e97-252edd2f1661",
   "metadata": {},
   "source": [
    "and there you have the it, all the scores you need.\n",
    "\n",
    "now if we want to dig into the results and figure out examples where your pipeline performed worse or really good you can easily convert it into a pandas array and use your standard analytics tools too!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bc72c682-b0c0-4314-9da5-9b22aae722a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>ground_truth</th>\n",
       "      <th>answer</th>\n",
       "      <th>contexts</th>\n",
       "      <th>faithfulness</th>\n",
       "      <th>answer_relevancy</th>\n",
       "      <th>context_recall</th>\n",
       "      <th>context_precision</th>\n",
       "      <th>harmfulness</th>\n",
       "      <th>answer_similarity</th>\n",
       "      <th>answer_correctness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the global implications of the USA Su...</td>\n",
       "      <td>The global implications of the USA Supreme Cou...</td>\n",
       "      <td>The global implications of the USA Supreme Cou...</td>\n",
       "      <td>[- In 2022, the USA Supreme Court handed down ...</td>\n",
       "      <td>0.958333</td>\n",
       "      <td>0.86077</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.940453</td>\n",
       "      <td>0.375738</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  What are the global implications of the USA Su...   \n",
       "\n",
       "                                        ground_truth  \\\n",
       "0  The global implications of the USA Supreme Cou...   \n",
       "\n",
       "                                              answer  \\\n",
       "0  The global implications of the USA Supreme Cou...   \n",
       "\n",
       "                                            contexts  faithfulness  \\\n",
       "0  [- In 2022, the USA Supreme Court handed down ...      0.958333   \n",
       "\n",
       "   answer_relevancy  context_recall  context_precision  harmfulness  \\\n",
       "0           0.86077             1.0                1.0            1   \n",
       "\n",
       "   answer_similarity  answer_correctness  \n",
       "0           0.940453            0.375738  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = result.to_pandas()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c08f613d-880c-4e13-9716-05ce23fa4576",
   "metadata": {},
   "source": [
    "And thats it!\n",
    "\n",
    "if you have any suggestion/feedbacks/things your not happy about, please do share it in the [issue section](https://github.com/explodinggradients/ragas/issues). We love hearing from you 😁"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
